feat: explorer mcp server for scenes#9339
Open
pravusjif wants to merge 90 commits into
Open
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t/explorer-mcp-server
Contributor
|
Windows and Mac build successful in Unity Cloud! You can find a link to the downloadable artifact below. |
pravusjif
added
force-build
Contributor
|
Warnings not reduced: 14668 => 14673 — remove at least one warning to merge. Warnings/errors in files changed by this PR (9)
|
…eat/explorer-mcp-server Resolved conflict in DynamicWorldContainer.cs: dropped the unused 'using CommunicationData.URLHelpers;' brought by the remote side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # Explorer/Assets/DCL/Infrastructure/Global/Dynamic/DynamicWorldContainer.cs
Centralize Formatting.Indented serialization in McpToolResult so tools can't drift the structured payload from its text mirror or forget the formatting. Convert the eight callsites and drop their now-unused Newtonsoft.Json import.
The builder produces both input and output JSON Schemas (OutputSchema, VectorSchema), so the "Input" name misrepresented it. Pure rename of the type and its file/test.
The three structured tools declare OutputSchema and build the payload by hand, so a field added to one and forgotten in the other would silently misdescribe the result. Add McpSchemaAssert.KeysMatch, a recursive schema-vs-payload key check, and wire it into the Get*/List* tool tests (populating a camera entity and a scene facade so nested objects are covered too).
… Build Dedup nullable type-token logic in McpJsonSchema via a private TypeToken(type, nullable) helper, dropping the (JToken)type cast in Property and sharing it with Object. Add symmetric OutputSchema validation in McpToolsRegistry.Build so a malformed output schema fails fast at registration instead of reaching the client.
Change the extension to take HttpStatusCode instead of int, casting once inside the method. Drops the (int) cast and statusCode: label from all seven call sites and makes the status intent type-safe.
Cap each tools/call at TOOL_CALL_TIMEOUT with a CancellationTokenSource linked to the server-lifetime token. A hung tool now returns a timeout error result instead of holding its HttpListenerContext open until shutdown. Distinguish shutdown cancellation (rethrow) from timeout (error result) via a when (ct.IsCancellationRequested) filter.
Tools declared "switch to the main thread yourself" and 15 of 16 opened with an identical `await UniTask.SwitchToMainThread(ct)`. Hop once in McpJsonRpcDispatcher.CallToolAsync instead and flip the IMcpTool contract to "invoked on the main thread". Readers with no other await become non-async (UniTask.FromResult); tools that offload to the thread pool keep their return-trip hops.
Tool calls are marshalled onto the main thread by the dispatcher, so the check-and-set runs without preemption before the first await. This clears the ShouldNotUseThreadingApiDirectly convention test for ScreenshotTool.
mikhail-dcl
requested changes
Jul 20, 2026
mikhail-dcl
left a comment
mikhail-dcl
left a comment
Collaborator
There was a problem hiding this comment.
I am mainly concerned about 2 things:
- Allocations, though I understand it's MCP, yet it's the runtime production code. Currently, they are out of control entirely
- The threading model in
McpHttpServer, please review it: currently, everything executes on the main thread
| /// JSON Schema of the tool arguments. Must be a JSON Schema object (type=object); build it with | ||
| /// <see cref="McpJsonSchema" /> so a malformed schema is caught at registration, not on first use. | ||
| /// </summary> | ||
| JObject InputSchema { get; } |
Collaborator
There was a problem hiding this comment.
For inheritors you could call McpJsonSchema.Object() directly in the interface and pass it further down to avoid repetitions in every class
|
|
||
| public UniTask<McpToolResult> ExecuteAsync(JObject arguments, CancellationToken ct) | ||
| { | ||
| int limit = Mathf.Clamp(arguments.GetInt("limit", DEFAULT_LIMIT), 1, MAX_LIMIT); |
Collaborator
There was a problem hiding this comment.
Instruct AI to go over every tool and introduce named constants for every hardcoded one which is used multiple times
| logBuffer.CopyTo(entries, sinceSeq, errorsOnly, limit); | ||
|
|
||
| var output = new StringBuilder(); | ||
| output.AppendLine($"latestSeq={logBuffer.LatestSeq} returned={entries.Count}"); |
Collaborator
There was a problem hiding this comment.
Don't interpolate in StringBuilder, it already has overloads for every primitive type to avoid allocations.
Validate every other implementation
| bool errorsOnly = arguments.GetString("severity", "all") == "error"; | ||
| long sinceSeq = arguments.GetLong("sinceSeq", -1); | ||
|
|
||
| var entries = new List<SceneLogBuffer.Entry>(limit); |
| result = downResult; | ||
| } | ||
|
|
||
| private bool TryResolveTarget(ref McpPointerClickIntent intent, World sceneWorld, out McpPointerClickResult result) |
Collaborator
There was a problem hiding this comment.
Under which circumstances sceneWorld != intent.SceneWorld ? I see that intent.SceneWorld is set after all other functions are called but the flow is not clear.
Wouldn't it be better to have a separate clean-up function instead if actions on the previously set world are required?
| /// JSON Schema of the tool arguments. Must be a JSON Schema object (type=object); build it with | ||
| /// <see cref="McpJsonSchema" /> so a malformed schema is caught at registration, not on first use. | ||
| /// </summary> | ||
| JObject InputSchema { get; } |
Collaborator
There was a problem hiding this comment.
Build() can be also moved to the upper class as it's a mandatory step anyway for every implementation
| public static McpJsonSchema Object() => | ||
| new (); | ||
|
|
||
| public McpJsonSchema String(string name, string? description = null, string[]? enumValues = null, bool required = false, bool nullable = false) => |
Collaborator
There was a problem hiding this comment.
Please fix required warnings to avoid additional warnings with linter
| private HttpListener? listener; | ||
|
|
||
| /// <summary>The localhost URL the server listens on, e.g. <c>http://127.0.0.1:8123/unity-explorer-mcp</c>.</summary> | ||
| public string EndpointUrl => $"http://127.0.0.1:{port}/{ENDPOINT_PATH}"; |
Collaborator
There was a problem hiding this comment.
Move it to DecentralandUrls?
| // and bypasses it, so cap the read itself into a fixed buffer to keep the body bounded regardless. | ||
| using (var reader = new StreamReader(context.Request.InputStream, Encoding.UTF8)) | ||
| { | ||
| var buffer = new char[MAX_BODY_BYTES + 1]; |
Collaborator
There was a problem hiding this comment.
It can be allocated on stack if you switch to sync API.
Here, you don't need async APIs: you can simply dedicate a thread to the HttpListener, and never leave it
| switch (context.Request.HttpMethod) | ||
| { | ||
| case "POST": | ||
| await HandlePostAsync(context, ct); |
Collaborator
There was a problem hiding this comment.
Every await without setting SynchronizationCotnext will return the flow to the main thread
Schema, parsing and responses now share the enum itself as the single
source of truth instead of string constants:
- McpWireEnum<T> caches snake_case wire names per enum type (reflection
once at static init, zero allocations afterwards)
- McpJsonSchema.Enum<T>() and JObjectExtensions.TryGetEnum() with an
allowed-subset overload for engine enums exposed only partially
- set_camera_mode/walk/click_entity parse straight into CameraMode,
MovementKind and ClickKind; screenshot/get_scene_logs/click_entity
button use small tool-local enums
- wire value "drone" renamed to "drone_view"; camera mode in responses
is now wire-cased too (was PascalCase ToString)
- invalid enum values return an error instead of silently defaulting
- tool catalog in docs no longer restates argument values — tools/list
is the authoritative contract; mcp-server-engineer agent doc updated
mikhail-dcl
requested changes
Jul 21, 2026
| /// Returns false when the body exceeds <see cref="MAX_BODY_BYTES" />; <paramref name="body" /> is | ||
| /// then empty and the request must be rejected. | ||
| /// </summary> | ||
| private static bool TryReadBodyWithinCap(HttpListenerRequest request, out string body) |
Collaborator
There was a problem hiding this comment.
This flow didn't become better: on the contrary, it now has more allocations. I also wonder if a chunk request is a real case, because the previous flow simply dropped oversized bodies
| // A main-thread tool completes on the main thread and UniTask continuations run inline on the | ||
| // completing thread; hop back so the envelope serialization here and the HTTP write in the | ||
| // transport never spend main-thread time. | ||
| if (PlayerLoopHelper.IsMainThread) |
Collaborator
There was a problem hiding this comment.
This piece of code relates to the domain above - the caller site. And the description clearly indicates it. It's not a big issue, but switching back to the thread pool should be done in the consumer - as it's required by it, this dispatcher should stay agnostic to it
CallToolAsync now owns both hops: it switches to the main thread before the tool (under the timeout token, so a stalled main thread surfaces as the regular tool timeout) and back to the thread pool after, making the existing post-tool SwitchToThreadPool the paired undo of its own switch rather than a favour to the transport. McpTool loses the ExecuteAsync/ExecuteCoreAsync template method that existed only for the hop; requiresMainThread becomes the public RequiresMainThread contract member the dispatcher reads. GetSceneLogsTool keeps its opt-out and still answers while the main thread is busy or paused.
McpPointerClickSystem's three-phase state machine (DOWN/WAIT_TICK/UP with six mutable in-flight fields) is gone. McpPointerEventSystem now delivers exactly one pointer event per McpPointerEventIntent, and ClickEntityTool composes a click from two intents: a press, then a release carrying a McpPressHandoff (world, entity, tick, press-frame hit) that keeps the release ordered onto a later scene tick, bound to the same world, and able to replay the press hit when the fresh ray no longer reaches the target. The intent is written once and never mutated; the release fallback policy is now explicit in the tool instead of buried in the system. Clicks complete one frame sooner (the extra frame was a phase-transition artifact; the tick gate is unchanged) and the click_entity wire contract is untouched. Tests: rewritten for the new API plus three previously uncovered release branches — press-hit replay when blocked, target destroyed mid-click, and the mid-click scene-reload guard.
The preempt → install → complete/abandon ritual duplicated between ClickEntityTool/McpPointerEventSystem and WalkTool/McpInputOverrideSystem now lives in one place: IMcpRequest<TResult> marks a request component and the McpRequest helper owns SendAsync (preempting a pending request), CompleteAndRemove (copy out, structural removal, then resolve the awaiter) and AbandonAsync (tool-side timeout cleanup). McpMovementOverride joins the contract via an AsyncUnit completion. McpPointerEventIntent is now a genuinely immutable request: readonly fields set through the constructor, AimPoint + HasExplicitAimPoint merged into a nullable Vector3, and the Deadline field is gone — the system-side deadline duplicated the tool-side .Timeout(), so the tool watchdog is now the single timeout. As a result click_entity times out after exactly timeoutSec (previously timeoutSec + 2s grace) and a timeout always surfaces as an MCP error instead of sometimes an in-band hit:false. Tests: McpRequestShould covers install, preemption, complete-and-remove and a missing completion; the deadline test is removed with the mechanism.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Embeds an MCP (Model Context Protocol) server inside the Explorer so a coding agent (Claude Code, Codex, …) can see a running SDK7 scene (screenshot, player/scene state, scene console logs) and drive it (teleport, move, walk, look, camera, pointer clicks, chat, emotes, reload) — closing the edit → reload → verify loop for scene development without a human in the middle.
The primary target is a retail build launched by Creator Hub (no Unity Editor, no Unity license on the creator's machine); the same server runs unchanged in the Editor for internal dev and on PR builds for QA.
Scope & blast radius
--mcp/--mcp-port <port>(accepted from the command line or a deep link). Default port 8123.FeatureId.MCP_SERVER(FeaturesRegistry, resolved asappArgs.HasFlag(MCP) || appArgs.HasFlag(MCP_PORT));DynamicWorldContainer.CreateAsyncaddsMcpServerPluginonly when the feature is enabled.HttpListenerbinds to127.0.0.1and is never reachable from the network.DCL.McpServerassembly + two folded folders); no#if UNITY_EDITOR— Editor and build behave identically.Architecture (brief)
DCL.McpServerasmdef rooted atExplorer/Assets/DCL/McpServer/, split intoCore/(protocol + transport + tool contract),Tools/(one class per tool),Components/,Systems/,Utils/,Tests/.Systems/folds intoDCL.PluginsandTests/intoDCL.EditMode.Testsvia.asmrefso they can reach code the standalone assembly doesn't reference.System.Net.HttpListener(Streamable-HTTP), Newtonsoft, protocol spec2025-06-18,toolscapability only.McpHttpServer(server + Origin validation) →McpJsonRpcDispatcher(initialize/ping/tools/list/tools/call) →IMcpTool.McpMovementOverride,McpPointerClickIntent) + aUniTaskCompletionSource, consumed and removed by aBaseUnityLoopSystem(McpInputOverrideSystemin the input group,McpPointerClickSystemin presentation) so synthetic input/clicks flow through the real pipelines. Each toolawaitsSwitchToMainThreaditself before reading ECS/Unity; heavy work (base64) hops back to the pool.Key decisions & rationale
Standalone embedded server — no integration with Unity's official MCP.
Unity's official MCP (in
com.unity.ai.assistant) is Editor-only by construction: its transport chain runs through a relay binary into the Unity Editor process, with no runtime component for a built player, and its extension API is discovered by Editor-startupTypeCachescanning that doesn't exist in a player. It is also license-gated (Unity 6+, Assistant package, AI-Gateway sign-in, a per-tier connection entitlement) — creators have none of these — and its ~51 tools target Editor asset/script/shader work with no input simulation and no built-player interaction at all. For our primary scenario (retail build, no Editor) integration is simply impossible. Owning the server is cheap because the MCP layer is a thin protocol adapter; all real capability (back-buffer capture, input injection, ECS reads) must live in-process regardless of who serves the protocol.Embedded MCP-over-HTTP because Creator Hub will drive multiple engines, not just Unity.
The product direction is a Creator Hub gateway: the coding agent talks to one endpoint (Creator Hub), which fans out as an MCP client to whichever engine's embedded server is running — Unity, Bevy, or Godot. For Hub to use one generic MCP client across all of them, every engine must expose the same MCP-over-HTTP surface. That requirement is what drove the transport choice and what set aside the parallel automation approaches:
ws://1473, Fleck) is Unity/web-specific — Bevy and Godot will never ship it, so routing Unity through CDP would force Hub to maintain a bespoke non-MCP client just for Unity. It stays a sibling network-monitor concern, not the automation transport.DclHarness/ClaudeIPCprototype is an Editor-only, reflection-based file channel bound to a running Unity Editor — no retail/multi-engine path at all.An embedded, standalone MCP server per engine is precisely what makes the multi-engine gateway possible.
Hand-rolled JSON-RPC — not the
ModelContextProtocolC# SDK; no batching.The SDK's most valuable piece (HTTP hosting) lives in
ModelContextProtocol.AspNetCore(Kestrel / Generic Host), which Unity doesn't run — we'd still have to write our ownHttpListenertransport and the main-thread marshalling. Its reflection/assembly-scan tool discovery and runtime schema generation are exactly the patterns that get fragile under IL2CPP / managed-code stripping in retail builds, whereas our explicit registration is IL2CPP-safe by construction. It would also drag inSystem.Text.Json+Microsoft.Extensions.*, conflicting with the codebase's Newtonsoft + own composition-root DI. We borrow the SDK's design (schema-from-signature, DI-injected params, annotation hints), not its code — the whole dispatcher is ~150 lines. We track spec 2025-06-18, which removed JSON-RPC batching, so a single-object dispatcher (no batch-array path) is spec-correct, not a gap.Tool annotations (
readOnlyHint/destructiveHint/idempotentHint/openWorldHint, spec 2025-06-18) are emitted intools/list— cheap, forward-compatible, and the machine-readable hook a future auth-gate of mutating tools (and profile filtering) will read.Structured output seam (
structuredContent+outputSchema) is built as a purely-additive result envelope and adopted only as an exemplar on the read-only state tools (get_player_state,get_scene_state,list_scene_entities) that already build a JObject; every other tool returns text only. Doing it now locks the output-field contract while zero external consumers depend on the prose format.Ecosystem sanity-check. A sweep of third-party Unity MCP servers + Anthropic's tool-design guidance confirmed the foundation is spec-aligned. Almost the entire ecosystem is Editor-tooling with a sidecar process; being fully in-process inside a running player is unusual and correct — it sidesteps the domain-reload churn that is the ecosystem's dominant pain by construction. Curated ~16 workflow-shaped tools avoid the tool-list flooding that degrades LLM tool selection.
Security model
127.0.0.1only.Originis localhost (drive-by / DNS-rebinding defense); requests without anOriginheader (CLI clients, curl) are allowed.execute-arbitrary-C#tool (a foot-gun for a retail build with no auth). The trigger to add a token is the first mutating dev-only tool.Tool catalog (16 tools)
Full arguments/returns and troubleshooting in
docs/mcp-automation.md.screenshot,get_player_state,get_scene_state,get_scene_logs,list_scene_entities,get_entity_details.teleport,move_to,walk,look_at,set_camera_mode,set_camera_pose,send_chat,reload_scene,trigger_emote,click_entity.Tests
EditMode tests (folded into
DCL.EditMode.Tests) cover the load-bearing routing plus the input bridge:McpJsonRpcDispatcherShould— known tool dispatch, unknown tool →-32602, notification (id == null) dropped.McpToolsRegistryShould—tools/listserialization,TryGet/ name-guard.McpToolResultShould— result / error / structured-output envelope.McpHttpServerShould,McpInputSchemaShould.McpPointerClickSystemShouldand per-tool tests (GetEntityDetailsToolShould,ListSceneEntitiesToolShould,GetPlayerStateToolShould,GetSceneStateToolShould).Deliberately NOT in this PR / follow-ups
Iteration-1 holes are intentional — the foundation is built to extend organically:
notifications/cancelledhandler + a per-request CTS registry land with the first genuinely long-running/interruptible tool.get_entity_details(char cap) andlist_scene_entities(limit + "narrow with X"); acomponent-name filter needs extendingIWorldInfo(out of scope).tools-only (noresourcescapability declared): read-only state that is spec-naturally a Resource (scene state, entity list, log buffer) stays a polling tool for now. Two narrow slices are worth adopting later —resource_linkin results (selectively; not forscreenshot, which agents consume best as an inline image) andresources/subscribe(the one real win over polling, adopted only when a consumer needs push and the target client supports it).verb_nounsnake_case; the only engine-specific risk isset_camera_modevaluesdrone/free, to be isolated behind a profile later).--mcp-profile creator|dev, auth token, per-tool toggles — designed, not built.Related: V2 issue #9393
How to enable / test
Endpoint:
http://127.0.0.1:<port>/unity-explorer-mcp. Connect an agent:Connection details:
docs/mcp-automation.md#connecting-a-coding-agent.Test Steps
[simpler SKILL pending for creators to use the MCP server in a more efficient way]
positionvalue):"C:\Users\[YOUR-USER]\Downloads\Decentraland_windows64\Decentraland.exe" --realm http://127.0.0.1:8000 --position 0,0 --local-scene true --skip-version-check true --mcpopen Decentraland.app --args --realm http://127.0.0.1:8000 --position 0,0 --local-scene true --skip-version-check true --mcpclaude mcp add --transport http --scope user explorer http://127.0.0.1:8123/mcp.